Database and system architecture for analyzing multiparty interactions

ABSTRACT

An analytics engine (AE) computing system for analyzing and evaluating data in real-time associated with a performance of parties interacting within a multi-party interaction is provided. The AE system is configured to receive interaction data from a data validation (DV) computing device, retrieve contextual data from a contextual data source, determine a task identifier, and calculate a task score. The AE system is also configured to retrieve normalization model data from a normalization database, compare a plurality of normalization rules to the validated interaction data and the contextual data, and determine at least one normalization factor applies to the task score. The AE system is further configured to normalize the task score based on the at least one normalization factor, calculate an aggregate score using the normalized task score, and store the validated interaction data, the normalized task score, and the aggregate score in an analysis database.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/063,468, filed Oct. 5, 2020, entitled “DATABASE AND SYSTEMARCHITECTURE FOR ANALYZING MULTIPARTY INTERACTIONS,” which issues asU.S. Pat. No. 11,507,564 on Nov. 22, 2022, which is a continuation ofU.S. patent application Ser. No. 15/694,462, filed Sep. 1, 2017,entitled “DATABASE AND SYSTEM ARCHITECTURE FOR ANALYZING MULTIPARTYINTERACTIONS,” which issued as U.S. Pat. No. 10,831,743 on Nov. 10,2020, which claims the benefit of priority to U.S. Provisional PatentApplication Ser. No. 62/382,997, filed Sep. 2, 2016, entitled “SYSTEMSAND METHODS FOR ANALYZING PERFORMANCE IN TEAM ENVIRONMENTS,” the entirecontents and disclosure of which are hereby incorporated by reference intheir entirety.

BACKGROUND

The field of the disclosure relates generally to analysis of multi-partyinteractions, and more particular, to networked-based systems andmethods for analyzing and evaluating data in real-time associated with aperformance of parties interacting within a multi-party interaction.

In some multi-party interactions, such as in sports, in competitions,and even in workplaces, statistics and other performance analytics areused to provide a quantitative measurement of a party or a multi-party'sperformance of a job or specific task. The quantitative measurement maybe used to supplement qualitative performance analysis of a party tobetter assess how well the party has performed a particular task.Performance analysis is typically performed by collecting dataassociated with a candidate party and analyzing the data to evaluateand/or rank particular tasks and other events from the data.

At least some multi-party interactions may be difficult to evaluate andquantify the performance of a party interacting within the multi-party.In some of these cases, only a portion of the data from the interactionis analyzed to generate performance statistics. This can greatly limitthe evaluation. These limited statistics may not capture the entirety ofthe party's performance and therefore the statistics may causemisleading conclusions to be prematurely made. For example, in sports,common statistics collected during a football game for each player mayonly be collected during some plays and may not represent a player'sperformance throughout the entire game. As such, the player'sperformance may be analyzed on a subset of the available data, thusproducing a limited assessment of the player's performance. In addition,when comparing a player's performance to other player's performance, itcan be difficult to evaluate many difficult variables associated withdifficult plays. In other words, comparing two different players fromdifferent teams, running different plays can be extremely difficult dueto the number of variable on each play even if both players playsubstantially the same position. The ability to normalize theseevaluations is needed. Accordingly, there is a need for providing aperformance analysis system having enhanced data acquisition, dataanalysis, and performance evaluation capabilities that is configured toanalyze in real-time a performance of a party and/or a multi-partyinteraction.

BRIEF DESCRIPTION

In one aspect, an analytics engine (AE) system for analyzing andevaluating data in real-time associated with a performance of partiesinteracting within a multi-party interaction is provided. The AE systemincludes at least one analytics engine (AE) computing device thatincludes a processor communicatively coupled to a memory and isconfigured to electronically receive validated interaction data from adata validation (DV) computing device, wherein the validated interactiondata includes at least a real-time data source identifier, a partyidentifier, task measurement data, and at least one category identifier.The AE computing device is also configured to retrieve contextual datafrom a contextual data source based on the party identifier in thevalidated interaction data wherein the contextual data includes at leastan interaction identifier. The AE computing device is further configuredto determine a task identifier, based at least in part on the partyidentifier, the interaction identifier, and the at least one categoryidentifier, calculate a task score using the contextual data and thetask measurement data wherein the task score is associated with the taskidentifier, and retrieve normalization model data from a normalizationdatabase based at least in part on the at least one category identifierwherein the normalization model data includes a plurality ofnormalization rules and a plurality of normalization factors. The AEcomputing device is also configured to compare the plurality ofnormalization rules to the validated interaction data and contextualdata, determine, based on the comparison, at least one normalizationfactor of the plurality of the normalization factors applies to the taskscore, normalize the task score based on the at least one normalizationfactor, calculate an aggregate score using the normalized task score,and store the validated interaction data, the normalized task score, andthe aggregate score in an analysis database based on the task identifierwherein the analysis database is partitioned based at least in part on aparty identifier and a task identifier.

In another aspect, a computer-implemented method for analyzing andevaluating data in real-time associated with a performance of partiesinteracting within a multi-party interaction is provided. The method isperformed using an analytics engine (AE) computing device that includesat least one processor in communication with at least one memory device.The method includes electronically receiving validated interaction datafrom a data validation (DV) computing device wherein the validatedinteraction data includes at least a real-time data source identifier, aparty identifier, task measurement data, and at least one categoryidentifier. The method also includes retrieving contextual data from acontextual data source based on the party identifier in the validatedinteraction data wherein the contextual data includes at least aninteraction identifier. The method further includes determining a taskidentifier, based at least in part on the party identifier, theinteraction identifier, and the at least one category identifier,calculating a task score using the contextual data and the taskmeasurement data wherein the task score is associated with the taskidentifier, and retrieving normalization model data from a normalizationdatabase based at least in part on the at least one category identifierwherein the normalization model data includes a plurality ofnormalization rules and a plurality of normalization factors. The methodalso includes comparing the plurality of normalization rules to thevalidated interaction data and the contextual data, determining, basedon the comparison, at least one normalization factor of the plurality ofthe normalization factors applies to the task score, normalizing thetask score based on the at least one normalization factor, calculatingan aggregate score using the normalized task score, and storing thevalidated interaction data, the normalized task score, and the aggregatescore in an analysis database based on the task identifier wherein theanalysis database is partitioned based at least in part on a partyidentifier and a task identifier.

In yet another aspect, a non-transitory computer readable medium thatincludes executable instructions for analyzing and evaluating data inreal-time associated with a performance of parties interacting within amulti-party interaction is provided. When the computer executableinstructions are executed by an analytics engine (AE) computing devicethat includes at least one processor in communication with at least onememory device, the computer executable instructions cause the AEcomputing device to electronically receive validated interaction datafrom a data validation (DV) computing device wherein the validatedinteraction data includes at least a real-time data source identifier, aparty identifier, task measurement data, and at least one categoryidentifier. The computer executable instructions also cause the AEcomputing device to retrieve contextual data from a contextual datasource based on the party identifier in the validated interaction datawherein the contextual data includes at least an interaction identifier.The computer executable instructions further cause the AE computingdevice to determine a task identifier, based at least in part on theparty identifier, the interaction identifier, and the at least onecategory identifier, calculate a task score using the contextual dataand the task measurement data wherein the task score is associated withthe task identifier, and retrieve normalization model data from anormalization database based at least in part on the at least onecategory identifier wherein the normalization model data includes aplurality of normalization rules and a plurality of normalizationfactors. The computer executable instructions also cause the AEcomputing device to compare the plurality of normalization rules to thevalidated interaction data and contextual data, determine, based on thecomparison, at least one normalization factor of the plurality of thenormalization factors applies to the task score, normalize the taskscore based on the at least one normalization factor, calculate anaggregate score using the normalized task score, and store the validatedinteraction data, the normalized task score, and the aggregate score inan analysis database based on the task identifier wherein the analysisdatabase is partitioned based at least in part on a party identifier anda task identifier.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is block diagram of an analytics engine (AE) system for analyzingand evaluating a data in real-time associated with a performance ofparties interacting within a multi-party interaction.

FIG. 2 is a data flow diagram of an example data analysis and evaluationprocess used by the AE system shown in FIG. 1 in accordance with anembodiment of the present disclosure.

FIG. 3 is an example configuration of an analysis database used by theAE system shown in FIG. 1 in accordance with an embodiment of thepresent disclosure.

FIG. 4 is a data flow diagram of an alternative example data analysisand evaluation process used by the AE system shown in FIG. 1 inaccordance with an embodiment of the present disclosure.

FIG. 5 illustrates an example configuration of user system, such as aclient computing device shown in FIG. 1 , in accordance with anembodiment of the present disclosure

FIG. 6 illustrates an example configuration of a host system for use inthe system shown in FIG. 1 in accordance with an embodiment of thepresent disclosure.

FIG. 7 is a flow diagram of an example method for analyzing andevaluating a data in real-time associated with a performance of partiesinteracting within a multi-party interaction performed by the AE systemshown in FIG. 1 .

FIG. 8A is a first example schematic diagram of a football play that maybe used for analyzing and evaluating a data in real-time associated witha performance of parties interacting within a multi-party interactionusing the system shown in FIG. 1 .

FIG. 8B is a second example schematic diagram of a football play thatmay be used for analyzing and evaluating a data in real-time associatedwith a performance of parties interacting within a multi-partyinteraction using the system shown in FIG. 1 .

FIG. 9 illustrates an example user interface for displaying base dataassociated with a football game.

FIG. 10 illustrates an example user interface for inputting base dataassociated with a football game.

FIG. 11A illustrates a first example user interface to display partydata associated with an interaction.

FIG. 11B illustrates a second example user interface to display partydata associated with an interaction.

DETAILED DESCRIPTION

The systems and methods described herein relate generally to analysis ofdata associated with a performance of one or more parties. Inparticular, the systems and methods described herein collect interactiondata including task measurements associated with one or more tasksperformed by one or more parties within a multi-party interaction,analyze the task measurements and contextual data to determine a taskscore in real-time for each task, normalize the task scores, andgenerate an aggregate score for the party and/or multiples partiesrepresenting a performance of the party and/or the multiple parties.

The systems and methods described herein include some examples of howthe systems and methods may be implemented. These examples are notintended to limit the systems and methods in any way. Rather, thesesystems and methods are used to analyze numerous multi-partyinteractions. One multi-party interaction that is described herein forexample purposes is American football (also referred to as “football”).However, it is to be understood that the multi-party interaction offootball is used for illustrative purposes only and is not intended tolimit the systems and methods described herein. For example, the systemsand methods described herein may be used to generate aggregate scoresfor other multi-party interactions, such as, but not limited to,workplaces, school, competitions (e.g., chess, debate competitions,etc.), and other sports (e.g., basketball, baseball, hockey, soccer,cricket, rugby, racing, etc.). As used herein, a “team member” refers toa party of a multi-party interaction. The team member has one or more“positions” or roles that have a set of “tasks” that the team memberperforms for the team. The position may be part of a “position group”that includes several related positions. The team members participate inan “multi-party interaction”, which may be separated into a plurality of“interactions.” In one example, the multi-party interaction is afootball game and the interactions are plays performed in the game. Insome embodiments, the multi-party interaction is a predetermined periodof time.

An analytics engine (AE) system analyzing and evaluating data inreal-time associated with a performance of parties interacting within amulti-party interaction is described herein. The AE system includes atleast one AE computing device, at least one real-time data source, andat least one client computing device, as described herein. The AEcomputing device includes one or more processors and a memory device incommunication with the processor for storing instructions. In someembodiments, the AE system includes multiple computing devicescommunicatively coupled together to perform the features andfunctionality of the AE computing device, such as a data validation (DV)computing device and a data interface (DI) computing device. The AEcomputing device is communicatively coupled to each client computingdevice and each real-time data source. In some embodiments, the clientcomputing devices are replaced with user interfaces that arecommunicatively coupled to the AE computing device. In otherembodiments, the client computing devices include the real-time datasources. In yet other embodiments, the real-time data sources may beapplication program interfaces (APIs), IP addresses, databases, and/orother type real-time data source that may transmit and receive data toand from the AE computing device. As defined herein, real-time relatesto the AE system processing data within a short period of time or as theuser is performing the tasks with a computing device (e.g., millisecondsto seconds, or possibly minutes depending upon the task) so that thedata output and/or input is available virtually immediately.

The AE computing device is configured to generate aggregate scores forone or more parties within a multi-party interaction, and display theaggregate scores to enable end-users to make decisions based on thedisplayed aggregate scores. In the example embodiment, the AE computingdevice is further configured to receive task measurements from thereal-time data sources that are used to generate at least in part theaggregate scores. In other embodiments, the AE computing device isconfigured to generate the aggregate scores without receiving taskmeasurements from the real-time data sources.

In the example embodiment, the AE computing device is configured todefine a task score for each task of a plurality of tasks. The tasks areassociated with a particular multi-party interaction that the AEcomputing device is configured to analyze. For example, using thefootball multi-party interaction, the plurality of tasks may includetasks, such as passing, rushing, receiving, blocking, and the like. Eachtask is associated with a set of predefined task measurements. A taskmeasurement is a discrete, quantitative measurement of the performanceof a task that the AE computing device uses to generate a task score andan aggregate score, as described herein. The task measurementsfacilitate capturing one or more parties' performance, includinginformation that is not typically included in traditional statistics fora particular multi-party interaction. In the example embodiment, each ofthe tasks is associated with at least one task measurement of zero thatis representative of ‘as expected’ or ‘normal’ performance. The tasksare further associated with negative and positive task measurements torepresent poor and good performance of the task, respectively. Forexample, the task measurements may be assigned on a scale of −2 to +2.In other embodiments, different ranges of task measurements may be used.For each task measurement, the AE computing device is configured toinclude a description data field that indicates what level ofperformance is representative of the task measurement, as describedherein. In some embodiments, the AE computing device is configured toreceive at least a portion of the task measurements from one or morereal-time data sources. The AE computing device is also configured togenerate task scores based on the received task measurements.

The AE computing device is further configured to store the taskmeasurements and task scores in a database communicatively coupled tothe processor. The database may store task measurements and task scoresin sections defined by related tasks to facilitate efficient look-upsand identification of tasks and scoring. The AE computing deviceprovides the real-time data sources access to the stored taskmeasurements and task scores. The AE computing device may transmit thestored task measurements and task scores to the real-time data sourcesand/or provide a communication interface that the real-time data sourcesaccess to view the stored task measurements and task scores. Users ofthe real-time data sources, such as collectors, may reference the storedtask measurements to input task measurements into the real-time datasources, as described herein.

In the example embodiment, the AE computing device is configured tocollect the task measurement data via interaction data transmitted by atleast one real-time data source. The interaction data is associated witha multi-party interaction occurring during a multi-party identifier andincludes, but is not limited to, an interaction identifier, a partyidentifier, a multi-party identifier, a location identifier, a positionidentifier, a timestamp, comment fields, normalization factors, taskmeasurement data including at least one task measurement, at least onetask measurement identifier, and a task identifier, and/or other formsof data that may be collected from the multi-party interaction. In theexample embodiment, the interaction data is associated with a playoccurring during a football game. The AE computing device is alsoconfigured to retrieve contextual data from a contextual data source.The contextual data includes, but is not limited to, event identifiers,event base data, party identifiers, and party base data. In someembodiments, contextual data includes any number of categoryidentifiers.

As used herein, base data refers to additional properties (e.g., datafields) may be associated with events, parties, interactions, and/ortasks. In some embodiments, base data is associated with a taskmeasurement (e.g., task base data), and is included in real time datareceived from a real time data source. In one embodiment, where the taskmeasurement is associated with a football player, the task base dataincludes an offense or defense category. In other embodiments, base datais associated with an interaction (e.g., interaction base data) and isretrieved from the contextual data source using an interactionidentifier. For example, interaction base data may include a playcategory, indicating a type of football play. (e.g., sweep, runningplay).

In the example embodiment, the AE computing device is configured toreceive user input in the form of interaction data from the real-timedata sources. For example, users accessing the real-time data sourcesmay review a video of a football game and assign task measurements toeach task performed by one party. In one example, each user hasparticular input tasks that the user may enter along with one partyand/or multiple parties that each user is responsible to assign taskmeasurements for at least a portion of the multi-party interaction. Inthe example embodiment, some users may specifically review tasks, suchas pre-snap formations, passing plays, running plays, or special teamsplays. The real-time data sources transmit the interaction data (e.g.,user input data) to the AE computing device. Once the interaction datais received by the AE computing device, the AE computing device parsesthe interaction data based on the contextual data, retrieves the taskmeasurements from the interaction data, and stores the task measurementsin an analysis database. Each task measurement is associated with aparty identifier corresponding to a particular party within themulti-party interaction. The task measurements may also be associatedwith comment fields where the users may input comments on how aparticular task measurement was determined. The comment fields areconfigured to store the comments in a particular format to enable the AEcomputing device to parse the comment field and identify the commentsprovided by the users. In some embodiments, the comments include anynumber of tags. In one embodiment, the AE computing device is configuredto identify tagged comments based on character strings, such as “TSFG”and “TFL.” Additionally or alternatively, comment tags may be prefixedby tag symbols, such as “#TSFG” and “$TFL.”

In some embodiments, the AE computing device is configured to track eachtask that each user is responsible to assign task measurements. The AEcomputing device tracks each task using the task identifier included inthe interaction data. By tracking each task, the AE computing device isable to determine if one or more task measurements are missing from theinteraction data. The AE computing device is able to determine whichtask measurements are missing by comparing the received taskmeasurements to predefined task measurements associated with the taskidentifier. If the AE computing device determines that one or more taskmeasurements are missing, the AE computing device transmits anotification message to the real-time data source that transmitted thetask measurements. The notification message may include data, such as atask measurement identifier associated with the missing taskmeasurement, a task identifier, a timestamp associated with the task,and other data that the real-time data source may require to identifythe task measurement that is missing. In certain embodiments, the AEcomputing device is further configured to determine if the one or moretask measurements are inconsistent or incorrect, such as by using a thedata validation (DV) computing device to compare multiple instances oftask measurements.

In at least some embodiments, when the AE computing device receivesinteraction data from a real-time data source, the AE computing deviceupdates the analysis database and transmits synchronously theinteraction data to other real-time data sources. For example, if onereal-time data source transmits interaction data for pre-snap formationsof a play in a football game, the AE computing device receives theinteraction data and updates the analysis database using at least oneparty identifier included in the interaction data. The updated analysisdatabase includes data fields for each party identifier to enable the AEcomputing to parse and/or filter data within the analysis database basedon the party identifier. By parsing and filtering data based on theparty identifier, the AE computing device is able to retrieve andtransmit output analysis data from the analysis database. The AEcomputing device is configured to transmit the output analysis data toclient computing devices associated with end-users and/or real-time datasources.

In the example embodiment, the AE computing device is in communicationwith the data validation (DV) computing device. In other embodiments,the AE computing device performs the functions of the DV computingdevice, as described herein. The DV computing device is configured toreceive the interaction data from multiple real-time data sources, andcompare the received interaction data, and more specifically, the taskmeasurements for one task associated with one or more parties in aninteraction. The DV computing device performs the comparison to verifythat the task measurements are consistent between real-time datasources. In the example embodiment, a first real-time data source (e.g.,real-time data source A) transmits a first task measurement for a task,and a second real-time data source (e.g., real-time data source A)transmits a second task measurement for the same task. The DV computingdevice is configured to compare the first and second task measurements(e.g., a comparison between the task measurements received fromreal-time data source A and B). If both scores match, the DV computingdevice determines that the task measurement for the task is verified,and stores the task measurement in the analysis database. However, ifthe task measurements do not match, The DV computing device isconfigured to identify the differences between the task measurements.The DV computing device is also configured to transmit the identifieddifferences to a third real-time data source, such as a computing devicethat is authorized to reconcile task measurement conflicts. The AEcomputing device transmits the task measurements and the identifieddifferences to the third real-time data source for review. In response,the third real-time data source may transmit to the DV computing devicea message including one of the two task measurements as the verifiedtask measurement or a message with a new task measurement. The AEcomputing device is further configured to store a comparison log in theanalysis database to maintain records of each instance that the DVcomputing device performs a comparison and/or identifies differences inthe comparison to facilitate tracking errors and improving datacollection.

The AE computing device is also configured to calculate a task score fora task performed by one party once the AE computing device receives theinteraction data. The AE computing device parses the interaction dataand retrieves the task measurements from the interaction data associatedwith the task. Subsequently, the AE computing device calculates the taskscore by performing an average of the retrieved task measurements. TheAE computing device may also calculate the task score by aggregating theretrieved task measurements. For example, the AE computing device maycalculate a task score for a running back by using task measurements,such as a rushing measurement, a blocking measurement, and a receivingmeasurement. The AE computing device stores the task scores within theanalysis database.

The AE computing device is further configured to normalize the taskscores received during a multi-party interaction to provide accuratemetrics for evaluating performance of a party in comparison to otherparties interacting in the multi-party interaction. To normalize thetask scores for a party, the AE computing device is configured todetermine whether a task and corresponding task measurements areassociated with normalization factors. The normalization factors aredata elements that indicate multi-party interaction conditions in whichthe task was performed. In the example embodiment, the normalizationfactors are included in the interaction data and are retrieved, by theAE computing device, from the interaction data. In other embodiments,the normalization factors are directly input by a user into the AEcomputing device. The AE computing device is configured to store thenormalization factors in a normalization database in communication withthe AE computing device. In one example, for a pass rusher, thenormalization factors may include down and distance, stance, positionrelative to the other team's players, quarterback drop (e.g.,three-step, seven-step, etc.), and so forth.

The AE computing device is configured to build normalization rules andgenerate normalization model data based on the normalization rules. TheAE computing device uses stored normalization factors to build thenormalization rules. The stored normalization factors may be receivedfrom past task measurement and base data received (e.g., historical dataover previous games and/or previous years of games) and/or inputdirectly into the AE computing device. The AE computing device is alsoconfigured to generate the normalization model data based on thenormalization rules and update the normalization model data once thenormalization factors are received. The normalization model dataincludes predefined values indicating an average task score for a taskand the normalization rules associated with each task. In the exampleembodiment, the AE computing device stores the normalization rules andthe normalization model data in the normalization database. By using thenormalization model data, the AE computing device may determine if atask score requires normalization. In one embodiment, the normalizationrules map category identifiers to normalization factors. For example, anormalization rule may include determining a normalization factor isassociated with interaction data based on a specific category identifierincluded in the interaction data and/or associated contextual data.Additionally or alternatively, the normalization rule may be compared totask measurement data.

In one example, the AE computing device may determine that a task scorerequires to be normalized to zero. A task score may be normalized tozero if the party associated with the task score had a limited chance ofhaving a positive or negative performance of the task. The AE computingdevice determines whether the party had a limited chance of having apositive or negative performance of the task by comparing the number oftask scores received from that party to the average number of taskscores received from other parties performing the same task during themulti-party interaction. In another example, the model normalizationdata for a pass rusher on a seven-step drop is typically greater thanthe model normalization data for a three-step drop (e.g., this isbecause typically a greater chance of a positive performance on aseven-step drop play versus a three-step drop play for a defensivepass-rusher). Therefore, the pass rusher's task score when thequarterback takes a seven-step drop is negatively affected bynormalization more than the pass rusher's score for a three-step drop.Accordingly, the AE computing device applies the normalization modeldata to task score with the same or similar task conditions as theconditions in which prior similar task were performed, thereby removingthe task condition advantage from the task score.

The AE computing device is also configured to store a list ofnormalization exceptions. In the example embodiment, the AE computingdevice receives the list of normalization exceptions from the real-timedata sources. The AE computing device is further configured to comparethe normalization rules to the list to determine if a normalizationexception has occurred. In another embodiment, the AE computing devicereceives a flag from one or more real-time data sources indicating thata normalization exception has occurred. If the AE computing devicedetermines that a normalization exception has occurred, the AE computingdevice does not normalize the task score.

The AE computing device may further normalizes the task scores by usinga replacement factor. The replacement factor is a task score of a partyrelative to an average task score of replacement parties that mayinteract in the same position (i.e., categorization) as the party withina multi-party (e.g., a team). The average task score of replacementparties are defined by averaging the task scores of parties that arefreely available to join the team. In the example embodiment, theaverage task score of replacement parties is normalized to zero.Accordingly, the replacement factor is the difference between the taskscore of the party and the average task score of replacement parties.

The AE computing device is configured to normalize a party's task scoreby, at least partially, using the normalization model data and thereplacement factors. In one embodiment, the AE computing devicegenerates a task score modifier as a function of the normalization modeldata and the replacement factors. In particular, the task score modifieris the sum of the normalization model data and the replacement factorsmultiplied by a number of tasks of a party during a multi-partyinteraction. The AE computing device is configured to generate theparty's normalized task score by subtracting the task score modifierfrom the party's task score. The AE computing device repeats thenormalization for each party's task score that the AE computing devicedetermines that a normalization exception does not apply.

Once the AE computing device has normalized the party's task score, theAE computing device calculate an aggregate score for the party in amulti-party interaction. In some embodiments, the AE computing devicegenerates the aggregate score by adding the party's normalized taskscores received during a multi-party interaction (e.g., a game). Inother embodiments, the AE computing device averages the party'snormalized task scores over the total number of tasks performed by theparty during the multi-party interaction. Similar to the taskmeasurements, each party may be associated with one or more aggregatescores.

The AE computing device is configured to use the party's normalized taskscore to calculate the aggregate score because using non-normalized taskscores to calculate the aggregate score may not accurately reflect aparty's performance in comparison to other parties within themulti-party interaction. For example, one party may have had moreopportunities to participate than other parties or may have had bettersituations to receive positive task scores than other parties. In oneexample, a pass rusher that participates on a relatively large number ofpassing plays with seven-step drops is likely to have a better taskmeasurement than a pass rusher that participates on a lower number ofsimilar passing plays.

In certain embodiments, the AE computing device is configured to changethe format or scale of the aggregate scores (e.g., 0-100 scale) fordisplay. In one example, the aggregate scores may be compared topredefined threshold values and assigned a letter grade or otherindicator based on the comparison. In embodiments in which the partiesare associated with a plurality of aggregate scores, the AE computingdevice is configured to apply weighting factors to each normalized taskscore when calculating an aggregate score associated with the party. Forexample, for an offensive tackle position, pass blocking has the highestweighting factor, run blocking has the next highest, and penalties havethe lowest weighting factor.

In at least some embodiments, the AE computing device uses the aggregatescore to generate a scaled aggregate score for a plurality ofmulti-party interactions. For example, the aggregate score for afootball game may be used to generate a scaled aggregate score for afootball season that includes the game. In some embodiments, theaggregate score may be adjusted to account for different scoringprocesses over time.

Once the AE computing device calculates the aggregate scores, the AEcomputing device may transmit the aggregate scores to client computingdevices, real-time data sources, and/or other computing devices to bedisplayed. In some embodiments, the aggregate scores are stored within adatabase, such as the analysis database. The analysis database isseparated into data tables, such as an event table, a party table, andinteraction table, and a task table. In the example embodiment, theevent table is configured to store data associated with a multi-partyinteraction (e.g., an event), such as an event identifier and event basedata. The party table is configured to store data associated with aparty of the multi-party interaction, such as a party identifier andparty base data. The interaction table is configured to store dataassociated with interactions, such as an interaction identifier,interaction base data, and interaction index value. The task table isconfigured to store data associated with a task such as taskmeasurements, a task identifier, and task scores. The analysis databaseis separated into data tables to facilitate storing event data and taskscores in a defined, searchable format. The analysis database isconfigured to be searchable and navigable to facilitate efficientlocation of particular data within the analysis database.

In at least some embodiments, the AE computing device is configured toidentify trends or other analytics using data stored in the analysisdatabase and normalization database. The AE computing device maytransmit the identified trends to real-time data sources for review. Incertain embodiments, the analysis database and the normalizationdatabase are in communication to enable the AE computing device toretrieve, calculate, transmit, and store data in real-time.

In some embodiments, the AE computing device receives a video includingvideo data and metadata from the client computing devices. The metadataidentifies a multi-party interaction associated with the video data(i.e., a game associated with the video data). In at least someembodiments, the metadata also includes markers that identify timestampswithin the video data for separate interaction of the multi-partyinteraction and other discrete portions or highlights of themulti-party-interaction. In certain embodiments, the metadata mayinclude predetermined statistics, party identifiers, positionidentifiers, metrics, and the like for the AE computing device to use.

The AE computing device is also configured to scan the video (e.g.,using image recognition techniques) to identify at least a portion ofthe video data and metadata, compare and match the video data andmetadata to data stored in the analysis database, and retrieve at leasta portion of the video data and metadata from the video data. In oneexample, the AE computing device may generate or modify particular datafields within the analysis database based on the metadata and/or thevideo data. For example, the AE computing device may scan the videoand/or parse the video data and metadata to automatically determine atype of play, field position, turnovers on downs, down and distance, andplayer roles and stances for a football game. In another example, the AEcomputing device may add and/or remove data from the video. That is, theAE computing device may add one or more task scores to the video andtransmit the video to a client computing device with instructions forthe client computing device to display the one or more task score in thevideo. In the example embodiment, the AE computing device is incommunication with the data interface (DI) computing device which isconfigured to generate an interface between the AE computing device andone or more client computing devices. The DI computing device may beconfigured to transmit and receive the video to and from one or moreclient computing devices. In other embodiments, the AE computing deviceperforms the functions of the DI computing device.

In the example embodiment, the AE computing device is configured toprovide each client computing device with access to at least a portionof the video data and data stored in the analysis database andnormalization database. In some embodiments, the AE computing device isconfigured to store a table of user permissions including end-useridentifiers. In one example, when a client computing device accesses theAE computing device, the AE computing device is configured to identifythe client computing device using an end-user identifier received fromthe client computing device. The AE computing device compares theend-user identifier to the end-user identifiers stored in the table ofpermissions, and determines which portions of the video may be displayedby the client computing device.

The methods and systems described herein may be implemented usingcomputer programming or engineering techniques including computersoftware, firmware, hardware or any combination or subset thereof,wherein the technical effects may be achieved by performing one of thefollowing steps: (i) receiving validated interaction data from a datavalidation (DV) computing device, wherein the validated interaction dataincludes at least a real-time data source identifier, a partyidentifier, task measurement data, and at least one category identifier;(b) retrieving contextual data from a contextual data source based onthe party identifier in the validated interaction data, wherein thecontextual data includes at least an interaction identifier; (c)determining a task identifier, based at least in part on the partyidentifier, the interaction identifier, and the at least one categoryidentifier; (d) calculating a task score using the contextual data andthe task measurement data, wherein the task score is associated with thetask identifier; (e) retrieving normalization model data from anormalization database based at least in part on the at least onecategory identifier, wherein the normalization model data includes aplurality of normalization rules and a plurality of normalizationfactors; (f) comparing the plurality of normalization rules to thevalidated interaction data and the contextual data; (g) determiningbased on the comparison, at least one normalization factor of theplurality of the normalization factors applies to the task score; (h)normalizing the task score based on the at least one normalizationfactor; (i) calculating an aggregate score using the normalized taskscore; and (j) storing the validated interaction data, the normalizedtask score, and the aggregate score in an analysis database based on thetask identifier, wherein the analysis database is partitioned based atleast in part on a party identifier and a task identifier.

The systems and methods described herein are configured to facilitate(a) improved performance analysis of interactions of a parties andmulti-parties; (b) improved measurements of data; (c) synchronizedanalysis for multiple client computing devices; (d) improvedverification of user input that includes task measurements; (e) improvedanalysis and evaluation speed of task measurement by implementing aspecific system architecture; and (f) reduced analysis time byautomating or partially automating a performance analysis process.

Described herein are computer systems such as a performance analyticscomputing device and a client computing device. As described herein, allsuch computer systems include a processor and a memory.

Further, any processor in a computer device referred to herein may alsorefer to one or more processors wherein the processor may be in onecomputing device or a plurality of computing devices acting in parallel.Additionally, any memory in a computer device referred to herein mayalso refer to one or more memories wherein the memories may be in onecomputing device or a plurality of computing devices acting in parallel.

As used herein, a processor may include any programmable systemincluding systems using micro-controllers, reduced instruction setcircuits (RISC), application specific integrated circuits (ASICs), logiccircuits, and any other circuit or processor capable of executing thefunctions described herein. The above examples are example only, and arethus not intended to limit in any way the definition and/or meaning ofthe term “processor.”

As used herein, the term “database” may refer to either a body of data,a relational database management system (RDBMS), or to both. As usedherein, a database may include any collection of data includinghierarchical databases, relational databases, flat file databases,object-relational databases, object oriented databases, and any otherstructured collection of records or data that is stored in a computersystem. The above examples are example only, and thus are not intendedto limit in any way the definition and/or meaning of the term database.Examples of RDBMS's include, but are not limited to including, Oracle®Database, MySQL, IBM® DB2, Microsoft® SQL Server, Sybase®, andPostgreSQL. However, any database may be used that enables the systemsand methods described herein. (Oracle is a registered trademark ofOracle Corporation, Redwood Shores, Calif.; IBM is a registeredtrademark of International Business Machines Corporation, Armonk, N.Y.;Microsoft is a registered trademark of Microsoft Corporation, Redmond,Wash.; and Sybase is a registered trademark of Sybase, Dublin, Calif.)

In one embodiment, a computer program is provided, and the program isembodied on a computer readable medium. In an example embodiment, thesystem is executed on a single computer system, without requiring aconnection to a sever computer. In a further embodiment, the system isbeing run in a Windows® environment (Windows is a registered trademarkof Microsoft Corporation, Redmond, Wash.). In yet another embodiment,the system is run on a mainframe environment and a UNIX® serverenvironment (UNIX is a registered trademark of X/Open Company Limitedlocated in Reading, Berkshire, United Kingdom). In certain embodiments,the system is run on a Linux® server environment (Linux is theregistered trademark of Linus Torvalds in the U.S. and other countries).The application is flexible and designed to run in various differentenvironments without compromising any major functionality. In someembodiments, the system includes multiple components distributed among aplurality of computing devices. One or more components may be in theform of computer-executable instructions embodied in a computer-readablemedium.

As used herein, an element or step recited in the singular and proceededwith the word “a” or “an” should be understood as not excluding pluralelements or steps, unless such exclusion is explicitly recited.Furthermore, references to “example embodiment” or “one embodiment” ofthe present disclosure are not intended to be interpreted as excludingthe existence of additional embodiments that also incorporate therecited features.

As used herein, the terms “software” and “firmware” are interchangeable,and include any computer program stored in memory for execution by aprocessor, including RAM memory, ROM memory, EPROM memory, EEPROMmemory, and non-volatile RAM (NVRAM) memory. The above memory types areexample only, and are thus not limiting as to the types of memory usablefor storage of a computer program.

The systems and processes are not limited to the specific embodimentsdescribed herein. In addition, components of each system and eachprocess can be practiced independent and separate from other componentsand processes described herein. Each component and process also can beused in combination with other assembly packages and processes.

FIG. 1 is a block diagram of an example analytics engine (AE) system 100for analyzing and evaluating data in real-time associated with aperformance of parties interacting within a multi-party interaction. AEsystem 100 includes any number of real-time data sources (e.g.,real-time data sources 142, 144, and 146) and data validation (DV)computing device 140. As used herein, real-time data sources includeWebSocket connections, HTTP connections, and the like, and is configuredto transmit interaction data, including task measurements, to DVcomputing device 140 such that interaction data is transmitted inreal-time. In some embodiments, a real-time data source includes acollector generating task measurements based on observations of areal-time multi-party interaction (e.g., a game). For example, real-timedata source 142 may include a WebSocket connection transmittinginteraction data to DV computing device 140 as the collector generatestask measurements for each task (e.g., play) performed by a party (e.g.,player). Additionally or alternatively, real-time data source 142includes an image processing computing device generating measurements.For example, real-time data source 144 may include a HTTP-based APIconnection transmitting interaction data to the DV computing device 140as the image processing computing device generates task measurements. Incertain embodiments, real-time data sources 142, 144, and 146 may beassociated with a source identifier, such as an IP address, username, orserial number. In one embodiment, source identifiers further include atrust level indicator, identifying a relative level of trust (e.g.,accuracy, precision, and reliability) associated with a real-time datasource. For example, a source identifier may indicate trust level 1 ortrust level A.

DV computing device 140 is configured to receive interaction data fromreal-time data sources, and transmit validated interaction data toanalytics engine (AE) computing device 110. In some embodiments, DVcomputing device 140 may be a component of AE computing device 110.

Analytics engine (AE) computing device 110 is communicatively coupled toDV computing device 140 and contextual data source 150. In someembodiments, AE computing device 110 is connected to DV computing device140 using a socket connection (e.g., Web Socket connection). AEcomputing device 110 is configured to receive validated interaction datafrom DV computing device 140, and to retrieve contextual data fromcontextual data source 150. In some embodiments, AE computing device 110includes a message queue, storing validated interaction data as it isreceived from the DV computing device 140. In the example embodiment, AEcomputing device 110 is communicatively coupled to normalizationdatabase 121 and analysis database 122. Normalization database 121stores normalization rules and normalization factors. Analysis database122 stores interaction data. In one embodiment, analysis database 122includes an event table, interaction table, party table, and task table.AE computing device 110 is configured to query (e.g., SQL queries, APIcalls) databases, including normalization database 121 and analysisdatabase 122. AE computing device 110 is further configured to receivequery responses.

Data interface (DI) computing device 120 is connected to, at least oneof, normalization database 121, analysis database 122, and AE computingdevice 110. In one embodiment, DI computing device 120 is configured toprovide an HTTP based API (i.e., Web API) to a client device, such asclient computing device 132. For example, DI computing device 120 mayreceive an API request including a party identifier from clientcomputing device 132, and DI computing device 120 may transmit an APIresponse including associated output analysis data as an XML file. Inanother embodiment, DI computing device 120 includes a web serverprocessing view requests. For example, DI computing device 120 mayreceive a view request from client computing device 132 (e.g., anend-user, such as the National Football League (NFL)), and may transmita HTML response over HTTP, including instructions to render a webpageincluding analysis information.

In other embodiments, AE system 100 may include additional, fewer, oralternative devices, including those described herein.

FIG. 2 is a data flow diagram of AE system 100 (shown in FIG. 1 ).Real-time data source 142 is configured to transmit interaction data210, including task measurement data, to DV computing device 140.Interaction data 210 includes a number of instances of task measurementdata. In some embodiments, interaction data 210 further includes areal-time data source identifier, such as an IP address, user name, ordevice type identifier. In one embodiment, task measurement dataincludes a data score, a time identifier (e.g., timestamp, relative timeindicator), and a measurement specification (e.g., measurement categoryidentifier, measurement unit). For example, task measurement data mayinclude a score of +1, a time remaining on a game clock (e.g., 05:14),and a category identifier associated with, for example, an analysis of afootball team's defense. Additionally or alternatively, task measurementdata may include a measurement data point and a unit. For example, taskmeasurement data may include a measurement of 450 yards, and a timeindicating that 5 minutes and 4 seconds remaining in the game. Incertain embodiments, task measurement data further includes a real-timedata source identifier, such as a username and/or IP address associatedwith the real-time data source. In some embodiments, task measurementdata includes category identifiers. Additionally or alternatively, AEcomputing device 110 is configured to determine category identifiersassociated with task measurement data, based on contextual data 214.

DV computing device 140 is configured to receive interaction data 210,process interaction data 210 into validated interaction data 212, andtransmit validated interaction data 212 to AE computing device 110. DVcomputing device 140 is configured to correlate task measurement databased on the data score and/or measurement specification (e.g., categoryidentifier, measurement unit). In some embodiments, DV computing device140 is further configured to correlate measurement data based onreal-time data source identifiers.

In some embodiments, DV computing device is configured to generatevalidated interaction data 212 based on measurement correlation.Measurement correlation includes determining the accuracy and/orprecision of data based on comparing multiple instances of taskmeasurement data included in interaction data 210. In one embodiment, DVcomputing device 140 is configured to generate validated interactiondata 212 based on receiving at least two substantially similar instancesof task measurement data (e.g., including similar measurementidentifiers, measurements, and timestamps) with differing sourceidentifiers. For example, DV computing device 140 may generate validatedmeasurement data based on two similar measurements provided fromdifferent real-time data sources, such as different collectors (e.g.,received task measurements from collectors A and B and compare themeasurements to identify inconsistencies). In certain embodiments, DVcomputing device 140 is be configured to detect measurement data faults.In one embodiment, DV computing device 140 is configured to identify atleast two instances of measurement data including similar measurementidentifiers and timestamps with differing measurements, and to generatean error notification, such as a debugging log entry, including the atleast two instances of measurement data.

Additionally or alternatively, DV computing device 140 is configured togenerate validated interaction data 212 based on a real-time data sourceidentifier. In some embodiments, DV computing device 140 is configuredto identify at least two instances of task measurement data includingsimilar measurement identifiers and timestamps with differingmeasurements and real-time data source identifiers. DV computing device140 is further configured to determine a trust level associated witheach task measurement data instance, based at least in part on thereal-time data source identifier. In one embodiment, the DV computingdevice determines a trust level associated with each real-time datasource identifier based on previous error notifications and/or adebugging log. For example, the DV computing device may determine, basedon the debugging log, a first real-time data source identifier is moreaccurate compared to a second real-time data source identifier, and mayfurther generate validated measurement data based on the measurementdata instances associated with the first real-time data sourceidentifier. In another embodiment, the real-time data source identifiermay include a trust level. For example, the real-time data sourceidentifier may include a calibration level, or accuracy rating, and/ortrust category, such as a real-time data source identifier indicating areal-time data source is associated with a high accuracy.

AE computing device 110 is also configured to receive validatedinteraction data 212, retrieve contextual data 214, store processedinteraction data 216, retrieve current normalization data 218, retrievestored interaction data 220, generate updated normalization data 219,and generate score data 217.

AE computing device 110 is configured to retrieve contextual data 214from contextual data source 150, based on validated interaction data212. Contextual data 214 includes event identifiers, event base data,party identifiers, and party base data. For example, contextual data 214may include team rosters, player names, teams/franchises, or the like.In some embodiments, contextual data 214 includes a player height,weight, and the like. Additionally or alternatively, contextual data 214includes combined metrics such as speed metrics (e.g., 40 yard dashtime), and performance metrics (e.g., yards per play, conversions,points per trip).

AE computing device 110 is configured to store processed interactiondata 216 in analysis database 122, based on validated interaction data212. AE computing device 110 is configured to process validatedinteraction data 212 into processed interaction data 216 for storage inanalysis database 122. In one embodiment, AE computing device 110generates database records and/or SQL queries based on validatedinteraction data 212. In some embodiments, AE computing device 110 isconfigured to determine at least one event identifier, and at least oneparty identifier associated with task measurement data included invalidated interaction data 212. In one embodiment, AE computing devicedetermines an event identifier and a plurality of party identifiersassociated with validated interaction data 212 based on the event basedata and/or party base data included in contextual data 214. In certainembodiments, AE computing device 110 is configured to generate databaseinstructions (e.g., SQL queries, API calls) to store validatedinteraction data 212 in analysis database 122. In one embodiment, AEcomputing device 110 is configured to parse validated interaction data212, determine identifiers from validated interaction data 212 based oncontextual data 214, and generate database instructions to storevalidated interaction data 212 and the determined identifiers inanalysis database 122. In some embodiments, processed interaction data216 includes instructions to store validated interaction data 212 as adata record in analysis database 122 having an interaction identifier,such that the stored data is accessible using any combination of theparty identifier, event identifier, and/or interaction identifier. Insome aspects, stored processed interaction data 216 is a parsed and/orindexed version of validated interaction data 212, including at leastthe task measurement data.

AE computing device 110 is configured to retrieve current normalizationdata 218 from normalization database 121 based on validated interactiondata 212 and contextual data 214. Current normalization data 218includes normalization factors and/or normalization rules associatedwith validated interaction data 212. In some embodiments, AE computingdevice 110 is configured to retrieve current normalization data 218,such as applicable normalization rules, based on the received contextualdata 214 associated with validated interaction data 212. For example, AEcomputing device 110 may retrieve current normalization data 218 basedon any combination of interaction identifier, player identifier, and/orevent identifier. Additionally or alternatively, AE computing device 110is configured to retrieve current normalization data 218 based on taskmeasurement data included in validated interaction data 212. Forexample, AE computing device 110 may retrieve normalization model data(i.e., current normalization data 218) associated with a category ofmeasurements. In one embodiment, the normalization factors include dataelements that indicate multi-party interaction conditions in which thetask was performed, and the normalization rules include instructionsconfigured to modify (e.g., normalize) task scores based on thenormalization factors and the analysis data.

AE computing device 110 is also configured to retrieve storedinteraction data 220 from analysis database 122, based on currentnormalization data 218. In some embodiments, AE computing device 110 isconfigured to retrieve stored interaction data 220 based onnormalization factors and/or normalization rules included in currentnormalization data 218. In one embodiment, where current normalizationdata 218 includes a normalization rule, the normalization rule mayinstruct AE computing device 110 to retrieve stored interaction data 220associated with validated interaction data 212, such as comparableand/or relevant interaction data. For example, relevant interaction datamay be used by the normalization rules to generate an average.Additionally or alternatively, AE computing device 110 may be configuredto retrieve stored interaction data 220 based on validated interactiondata 212 and contextual data 214. In some embodiments, AE computingdevice 110 retrieves stored interaction data 220 by querying analysisdatabase 122 using contextual data 214 with any combination of partyidentifiers, event identifiers, and/or interaction identifiers.

AE computing device 110 is configured to generate updated normalizationdata 219 based on validated interaction data 212, contextual data 214,and current normalization data 218. AE computing device 110 isconfigured to generate and/or update normalization data, such asnormalization rules and normalization model data. In some embodiments,the AE computing device 110 generates updated normalization rules byapplying the normalization model data to validated interaction data 212.For example, AE computing device 110 may generate updated normalizationdata 219 and store updated normalization data 219 in normalizationdatabase 121.

AE computing device 110 is configured to generate score data 217,including normalized task scores, and aggregated scores, based onvalidated interaction data 212, contextual data 214, stored interactiondata 220, and current normalization data 218. AE computing device 110 isconfigured to parse task scores from validated interaction data 212. Insome embodiments, AE computing device 110 is configured to generatenormalized task scores by applying normalization rules to task scores.In one embodiment, AE computing device 110 generates a task scoremodifier as a function of the normalization model data and thereplacement factors. In some embodiments, where real-time data includesnormalization exception information, AE computing device 110 isconfigured to disable normalization for a specific task score. In someembodiments, AE computing device 110 generates normalized task scoresbased on replacement factors. That is, AE computing device 110 generatesthe normalized task scores based on the task score of a party relativeto an average task score of replacement parties that may have a similarcategorization as the party. In some embodiments, AE computing device110 generates an aggregate score by adding the party's normalized taskscores received during a multi-party interaction (e.g., a game). Inother embodiments, the AE computing device 110 averages the party'snormalized task scores over the total number of tasks performed by theparty during the multi-party interaction, such that each party may beassociated with one aggregate score per multi-party interaction. AEcomputing device 110 is further configured to store score data 217 inanalysis database 122.

FIG. 3 depicts an example configuration of analysis database 122,included in AE system 100 (shown in FIG. 1 ). Analysis database 122includes, at least, event table 310, party table 320, interaction table330, and task table 340. Event records in event table 310 are uniquelyidentified by an event identifier 312. Party records in party table 320are uniquely identified by a party identifier 322. Interaction recordsin interaction table 330 are uniquely identified by an interactionidentifier 332. In some embodiments, identifiers are generated byanalysis database 122. Additionally or alternatively, identifiers may begenerated by AE computing device 110 (shown in FIG. 1 ) based oncontextual data 214 (shown in FIG. 2 ). In the example embodiment, taskrecords in task table 340 are uniquely identified by a combination of aninteraction identifier 342 and a party identifier 344. In an alternateembodiment, task records include a unique task identifier. Interactionrecords in interaction table 330 are associated with an event record inevent table 310, using an event identifier 334, such that an event isassociated with a set of interactions. Task records in task table 340include an interaction identifier 342, such that an interaction isassociated with a set of tasks. Task records further include partyidentifier 344, such that task records are associated with at least oneparty.

AE computing device 110 is configured to generate interaction records ininteraction table 330 based on validated interaction data 212 (shown inFIG. 2 ) and associated contextual data 214. Additionally oralternatively, AE computing device 110 is configured to determine aninteraction identifier 332 associated with validated interaction data212 and/or associated contextual data 214. In some embodiments, AEcomputing device 110 is configured to store at least part of contextualdata 214 as interaction base data 336.

AE computing device is also configured to generate task records in tasktable 340, based on task measurement data included in validatedinteraction data 212. Additionally or alternatively, AE computing device110 is configured to determine a task identifier and/or a partyidentifier 344 associated with validated interaction data 212 and/orassociated contextual data 214. In some embodiments, AE computing device110 is configured to store instances of task measurement data, includedin validated interaction data 212 (shown in FIG. 2 ), as records in tasktable 340. In one embodiment, AE computing device 110 stores an instanceof task measurement data in task base data 346, and determines anassociated interaction identifier 342 and party identifier 344 based onthe associated validated interaction data 212 and contextual data 214.In some embodiments, task base data 346 includes multiple instances oftask measurement data.

AE computing device 110 is configured to generate and/or create partydata records in event table 310, based on validated interaction data 212and/or associated contextual data 214. In some embodiments, where the AEcomputing device determines, based on contextual data 214, thatvalidated interaction data 212 is associated with at least one party,the AE computing device is configured to perform at least one ofretrieving a party identifier 322, or generating a party recordincluding party identifier 322. For example, validated interaction data212 may include a task measurement associated with a newly identifiedparty, and AE computing device 110 may generate a party record includingparty identifier 322 and party base data 324, based on contextual data214. Additionally or alternatively, party identifier 322 may begenerated by analysis database 122. In one embodiment, AE computingdevice 110 is configured to generate a party identifier 322 and partybase data 324 based on the first time a party identifier is identifiedin validated interaction data 212. AE computing device 110 is furtherconfigured to retrieve party identifiers based on subsequenceidentification of the party identifier in validated interaction data212. In certain embodiments, AE computing device 110 maintains a partytable 320 including unique records for each party that the AE computingdevice has analyzed across interactions and/or events.

AE computing device 110 is configured to generate and/or create eventdata records based on contextual data 214. In some embodiments, wherethe AE computing device determines contextual data 214 is associatedwith an event, the AE computing device is configured perform at leastone of, retrieving an event identifier 312 from event table 310, orgenerating an event record, including an event identifier 312. Forexample, contextual data 214 may include an event specification, and AEcomputing device 110 may be configured to generate an event record,including an event identifier 312 and event base data 314, based oncontextual data 214. As another example, AE computing device may beconfigured to retrieve an event identifier 312 based on an eventspecification included in contextual data 214. In certain embodiments,AE computing device maintains an event table 310 including uniquerecords for each set of interactions (e.g., event) analyzed.

In the example embodiment, AE computing device 110 is configured tostore task score data in analysis database 122. In some embodiments,where the AE computing device 110 generates a score (e.g., task score,normalized score) associated with a task, the AE computing device 110 isconfigured to transmit score data 217 to analysis database 122. Scoredata 217 includes, at least one score (e.g., task score, normalizedscore, scaled score, and the like) and an interaction identifier, suchas interaction identifier 342. Additionally or alternatively, score data217 includes a party identifier 344. Analysis database 122 is configuredto store at least part of score data 217 as task score data 348 in tasktable 340 using interaction identifier 342 and/or party identifier 344.

Analysis database 122 is configured to receive queries, and generatequery responses. In some embodiments, queries include any combination ofevent identifiers, party identifiers, and interaction identifiers.Additionally or alternatively, queries may include ranges and/or rulesfor selecting identifiers. Analysis database 122 is configured to filterbase data (e.g., event base data 314, party base data 324, interactionbase data 336, and task base data 346) based on the query, and generatea query response including the filtered data. For example, analysisdatabase 122 may generate a query response including task base data 346associated with a party identifier 344 included in the query.

In some embodiments, where task measurements include categoryidentifiers, AE computing device 110 is configured to store taskmeasurement data in interaction addendum data file 350. In oneembodiment, AE computing device 110 is configured to determine acategory identifier 354 associated with task measurement data includedin validated interaction data 212. For example, a task measurement mayinclude a category identifier (e.g., category identifier 354 and/orcategory identifier 366). In one embodiment, the AE computing device isconfigured to generate interaction addendum data file 350 in response toreceiving task measurement data including a category identifier, such ascategory identifier 354.

In certain embodiments, AE computing device 110 is configured togenerate task addendum data file 360 based on task measurement data, anda determined category identifier (e.g., category identifier 366), wherethe category identifier identifies a category of tasks. In an alternateembodiment, AE computing device 110 is configured to generateinteraction addendum data file 350 based on task measurement data, and adetermined category identifier (e.g., category identifier 354), wherethe category identifier identifies a category of interactions. Incertain embodiments, AE computing device 110 is configured to generateboth interaction addendum data file 350 and task addendum data file 360.In at least some embodiments, AE computing device 110 is configured todetermine if a category identifier is associated with a category oftasks and/or interactions.

In one aspect, AE computing device 110 is configured to accommodatestoring category-specific task measurements using addendum data, suchthat differing categories of tasks may have a partially consistent datastructure. For example, all categories of tasks may have a task record,and an associated addendum data file based on a category identifier. Insome embodiments, AE computing device 110 is configured to store anynumber of instances of task measurement data as task addendum data file360. Specifically, AE computing device 110 may generate a task addendumdata file 360, including the party identifier 362 and the interactionidentifier 364 of the associated task record, such that task addendumdata file 360 is associated with a task record included in task table340. Task addendum data file 360 further includes the determinedcategory identifier 366, and any number of instances of task measurementdata as task addendum data. In certain embodiments, category specifictask measurements are stored in task addendum data file 360, such asmeasurements associated with a specific type of interaction.

In alternative embodiments, AE computing device 110 is configured tostore any number of instances of task measurement data in interactionaddendum data file 350. Specifically, AE computing device 110 maygenerate an interaction addendum data file 350 including the interactionidentifier, such as interaction identifier 352, of the associatedinteraction record, such that the interaction addendum data file 350 isassociated with an interaction record from interaction table 330.Interaction addendum data file 350 further includes the determinedcategory identifier 354, and any number of instances of task measurementdata. In certain embodiments, category specific task measurements arestored in interaction addendum data file 350, such as measurementsassociated with a specific type of task.

In certain embodiments, AE computing device 110 is configured to storevideo data and/or video metadata as interaction addendum data. In oneembodiment, validated interaction data 212 includes video data and videometadata associated with an interaction. For example, AE computingdevice may determine a task identifier associated with a taskmeasurement, and may further store the video data as task addendum dataincluding the task identifier. Video data includes visual media in amachine readable format, such as MPEG4, WEBM, and the like. Videometadata includes, in one embodiment, a marker including video timestampindicating a location within the associated video, and an interactionidentifier associated with the marker.

FIG. 4 is a data flow diagram of AE system 100 (shown in FIG. 1 ). AEsystem 100 includes any number of client computing devices, such asclient computing devices 132, 401, and 402). In one embodiment, clientcomputing devices 132 and 401 is communicatively coupled to DI computingdevice 120. In an alternative embodiment, client computing device 402 iscommunicatively coupled to analysis database 122. In at least someembodiments, client computing devices 132, 401, and 402 may becommunicatively coupled to both analysis database 122 and DI computingdevice 120. DI computing device 120 is configured to respond to arequest (e.g., view request, API requests, queries) received from clientcomputing devices 132, 401, and/or 402. DI computing device 120 isfurther configured to generate and transmit a response (e.g., view data,output analysis data, API responses) to client computing devices 132,401, and/or 402. In some embodiments, DI computing device 120 may be acomponent of AE computing device 110.

In certain embodiments, output analysis data includes any combination oftask records, interaction records, interaction addendum data, taskaddendum data, party records, and event records. Output analysis datamay include full records, or parts of records. In one embodiment, outputanalysis data includes task identifiers and normalized task scores. Incertain embodiments, output analysis data further includes videometadata associated with video data, such that the output analysis datamay be correlated with a field location within the video data. In oneembodiment, output analysis data includes an interaction identifier andvideo metadata indicating a portion of the video data associated withthe interaction identified by the interaction identifier.

In some embodiments, DI computing device 120 is configured to provide awebsite to client computing devices 132, 401, and 402. In certainembodiments, DI computing device 120 is configured to receive a viewrequest 404 from client computing device 132. View request 404 includesany combination of event identifiers, task identifiers, partyidentifiers, and interaction identifiers, such that analysis dataassociated with the identifiers may be retrieved. Additionally oralternatively, view request 404 may include a request for recentlygenerated content, recently updated content, an index of availableanalysis data, and the like. In certain embodiments, DI computing device120 is configured to generate view data 406 based on view request 404.In one embodiment, DI computing device 120 retrieves output analysisdata from analysis database 122, and generates view data (e.g., HTML,CSS, JavaScript) based at least in part on the output analysis data.

In some embodiments, DI computing device 120 queries analysis database122 based at least in part on view request 404, to retrieve outputanalysis data. In certain embodiments, DI computing device 120 storesview templates (e.g., HTML templates, JavaScript templates) used toformat output analysis data as a webpage. For example, DI computingdevice 120 may generate a webpage based on output analysis data, andtransmit the webpage to client computing device 132.

In some embodiments, DI computing device 120 is configured to provide anAPI to client computing devices 132, 401, and/or 402. In certainembodiments, DI computing device 120 is configured to receive an APIrequest 408 from client computing device 401, and to retrieve outputanalysis data 410 from analysis database 122 based at least in part onAPI request 408. For example, DI computing device 120 may query analysisdatabase 122 based on an identifier included in API request 408. DIcomputing device 120 is configured to transmit output analysis data 410as an API response to client computing device 401. In certainembodiments, output analysis data 410 may include a HTTP transmission,further including any of a XML file, JSON file, HTML file, and the like.

In some embodiments, analysis database 122 is configured to receivequeries, such as queries 412, from client computing device 402. In oneembodiment, query 412 includes any combination of task identifiers,party identifiers, interaction identifiers, and event identifiers. Incertain embodiments, analysis database 122 is configured to retrieveoutput analysis data 414 based on query 412, and transmit outputanalysis data 414 to client computing device 402.

FIG. 5 depicts an exemplary configuration of a remote or computingdevice 502, such as client computing device 132 (shown in FIG. 1 ).Computing device 502 may include a processor 505 for executinginstructions. In some embodiments, executable instructions may be storedin a memory area 510. Processor 505 may include one or more processingunits (e.g., in a multi-core configuration). Memory area 510 may be anydevice allowing information such as executable instructions and/or otherdata to be stored and retrieved. Memory area 510 may include one or morecomputer-readable media.

Computing device 502 may also include at least one media outputcomponent 515 for presenting information to a user 530 (e.g., anend-user or a real-time data source). Media output component 515 may beany component capable of conveying information to user 530. In someembodiments, media output component 515 may include an output adapter,such as a video adapter and/or an audio adapter. An output adapter maybe operatively coupled to processor 505 and operatively coupled to anoutput device such as a display device (e.g., a liquid crystal display(LCD), organic light emitting diode (OLED) display, cathode ray tube(CRT), or “electronic ink” display) or an audio output device (e.g., aspeaker or headphones). In some embodiments, media output component 515may be configured to present an interactive user interface (e.g., a webbrowser or client application) to user 530.

In some embodiments, computing device 502 may include an input device520 for receiving input from user 530. Input device 520 may include, forexample, a keyboard, a pointing device, a mouse, a stylus, a touchsensitive panel (e.g., a touch pad or a touch screen), a camera, agyroscope, an accelerometer, a position detector, and/or an audio inputdevice. A single component such as a touch screen may function as bothan output device of media output component 515 and input device 520.

Computing device 502 may also include a communication interface 525,which may be communicatively coupled to a remote device, such as DIcomputing device 120. Communication interface 525 may include, forexample, a wired or wireless network adapter or a wireless datatransceiver for use with a mobile phone network (e.g., Global System forMobile communications (GSM), 3G, 4G or Bluetooth) or other mobile datanetwork (e.g., Worldwide Interoperability for Microwave Access (WIMAX)).

Stored in memory area 510 are, for example, computer-readableinstructions for providing a user interface to user 530 via media outputcomponent 515 and, optionally, receiving and processing input from inputdevice 520. A user interface may include, among other possibilities, aweb browser and client application. Web browsers enable users 530 todisplay and interact with media and other information typically embeddedon a web page or a website from a web server associated with anadministrator of AE system 100 (shown in FIG. 1 ). A client applicationallows users 530 to interact with a server application associated with,for example, AE computing device 110 (shown in FIG. 1 ).

FIG. 6 depicts an exemplary configuration of a host computing device602, such as AE computing device 110 and DI computing device 120 (shownin FIG. 1 ). Host computing device 602 may include a processor 605 forexecuting instructions. Instructions may be stored in a memory area 610,for example. Processor 605 may include one or more processing units(e.g., in a multi-core configuration).

Processor 605 may be operatively coupled to a communication interface615 such that host computing device 602 may be capable of communicatingwith a remote device such as computing device 502 (shown in FIG. 5 ) oranother host computing device 602. For example, communication interface615 may receive requests from computing device 502 via the Internet.

Processor 605 may also be operatively coupled to a storage device 625.Storage device 625 may be any computer-operated hardware suitable forstoring and/or retrieving data. In some embodiments, storage device 625may be integrated in host computing device 602. For example, hostcomputing device 602 may include one or more hard disk drives as storagedevice 625. In other embodiments, storage device 625 may be external tohost computing device 602 and may be accessed by a plurality of hostcomputing devices 602. For example, storage device 625 may includemultiple storage units such as hard disks or solid state disks in aredundant array of inexpensive disks (RAID) configuration. Storagedevice 625 may include a storage area network (SAN) and/or a networkattached storage (NAS) system.

In some embodiments, processor 605 may be operatively coupled to storagedevice 625 via a storage interface 620. Storage interface 620 may be anycomponent capable of providing processor 605 with access to storagedevice 625. Storage interface 620 may include, for example, an AdvancedTechnology Attachment (ATA) adapter, a Serial ATA (SATA) adapter, aSmall Computer System Interface (SCSI) adapter, a RAID controller, a SANadapter, a network adapter, and/or any component providing processor 605with access to storage device 625.

Memory areas 510 (shown in FIG. 5 ) and 610 may include, but are notlimited to, random access memory (RAM) such as dynamic RAM (DRAM) orstatic RAM (SRAM), read-only memory (ROM), erasable programmableread-only memory (EPROM), electrically erasable programmable read-onlymemory (EEPROM), and non-volatile RAM (NVRAM). The above memory typesare for example only, and are thus not limiting as to the types ofmemory usable for storage of a computer program.

FIG. 7 is a flow diagram of an example method 700 for analyzing andevaluating data in real-time associated with a performance of partiesinteracting within a multi-party interaction. In the example embodiment,at least a portion of method 700 is performed by an analysis engine (AE)computing device, such as analysis engine (AE) computing device 110,shown in FIG. 1 . In certain embodiments, method 700 may be at leastpartially performed by another computing device. In other embodiments,method 700 includes additional, fewer, or alternative steps, includingthose described herein.

Method 700 begins with the AE computing device electronically receiving702 validated interaction data from a data validation (DV) computingdevice, wherein the validated interaction data includes at least areal-time data source identifier, a party identifier, task measurementdata, and at least one category identifier. Method 700 also includesretrieving 704 contextual data from a contextual data source based onthe party identifier in the validated interaction data, wherein thecontextual data includes at least an interaction identifier. Method 700further includes determining 706 a task identifier, based at least inpart on the party identifier, the interaction identifier, and the atleast one category identifier, calculating 708 a task score using thecontextual data and the task measurement data, wherein the task score isassociated with the task identifier, and retrieving 710 normalizationmodel data from a normalization database based at least in part on theat least one category identifier, wherein the normalization model dataincludes a plurality of normalization rules and a plurality ofnormalization factors. Method 700 also includes comparing 712 theplurality of normalization rules to the validated interaction data andthe contextual data, determining 714, based on the comparison, at leastone normalization factor of the plurality of the normalization factorsapplies to the task score, normalizing 716 the task score based on theat least one normalization factor, calculating 718 an aggregate scoreusing the normalized task score, and storing 720 the validatedinteraction data, the normalized task score, and the aggregate score inan analysis database based on the task identifier, wherein the analysisdatabase is partitioned based at least in part on a party identifier anda task identifier.

FIGS. 8A and 8B are example schematic diagrams of two interactions,including parties, associated with an event. Specifically, the eventincludes interaction 800 and interaction 801. The event further includesparty 820, party 840, and party 860.

FIG. 8A is an example schematic diagram of interaction 800, includingparties 820, 840, and 860. Interaction 800 is included in a set ofinteractions, or an event. Task measurement 821 is associated with party820, and indicates a position and direction. Similarly, task measurement841 is associated with party 840, and task measurement 861 is associatedwith party 860. Parties 820 and 840 may be associated with an offensecategory identifier, and party 860 may be associated with a defensecategory identifier. In certain embodiments, AE computing device 110(shown in FIG. 1 ) is configured to receive interaction data includingmultiple instances of task measurements, such as interaction data 210 orvalidated interaction data 212 (shown in FIG. 2 ). In one embodiment, AEcomputing device 110 receives interaction data including taskmeasurement 821, task measurement 841, and task measurement 861, and theAE computing device 110 is configured to determine a party (e.g., partyidentifier) associated with each task measurement.

FIG. 8B is an example schematic diagram of interaction 801, includingparties 820, 840, and 860 (shown in FIG. 8A). Interaction 801 may besubsequent to interaction 800, and may further be associated with thesame event (e.g., associated with the same event identifier).Interaction 801 includes task measurement 842 associated with party 840.In certain embodiments, AE computing device 110 may be configured todetermine a category (e.g., category identifier) associated withinteraction 801 based on task measurement 842. For example, AE computingdevice 110 may store task measurement 842 as interaction addendum dataassociated with the identifier of interaction 801 in analysis database122.

In the example embodiment, during the analysis data collection process,AE computing device 110 or analysts associated with client computingdevices 132 (both shown in FIG. 1 ) analyze a video data file to assigntask scores for each player on offense (e.g., party 840) and defense(e.g., party 860). For the offensive line players, a task score isassigned for their performance of their respective blocking assignment.Table 1 is an example list of abbreviations that are used for reference.Additional abbreviations may be apparent to those skilled in the art,such as abbreviations associated with a specific task or event. Table 2includes an example performance criterion for a play-side tackle (PST)for the outside zone running play. In the example embodiment, theperformance criteria includes five predefined task scores within anegative one to positive one score range that can be assigned to theplay-side tackle for the play. The middle task score is a zero and isindicative of ‘as expected’ or ‘normal’ performance of the blockingassignment. Accordingly, the negative scores indicate a poor performanceof the blocking assignment and the positive scores indicate a goodperformance. Each task score includes a description that defines whatlevel of performance corresponds to the task score. In otherembodiments, a different range of task scores, number of task scores,and/or different definitions of the task scores may be used.

TABLE 1 Abbreviation Full Name QB Quarterback RB Running Back PS-TEPlay-Side Tight End PST Play-Side Tackle PSG Play-Side Guard C CenterBSG Back-Side Guard BST Back-Side Tackle BS-TE Back-Side Tight End RILBRight Inside Linebacker

TABLE 2 Position Responsibility −1 Play −0.5 Play 0 Play 0.5 Play 1 PlayPST Combo with Surrenders Fails to Controls Controls Drives 5- PSG;5-tech immediate sustain 5-tech on play-side tech on to RILB penetrationblock on combo should of combo to threaten 5-tech or with PSG 5-tech towith PSG ball carrier control on to allow sustain back to at or exchangerelease to play-side LBs or behind to allow RILB gap; On drives LOSrelease to release RILB at RILB; climbs to second Allows control levelout 5-tech or RILB & of play to RILB to maintain threaten threatenplay-side integrity ball gap at of defense carrier second just pastlevel LOS

With respect to FIG. 1 , AE computing device 110 is configured togenerate and update an analysis data file based on the received gamedata. In particular, the analysis data file includes at least a portionof the collected base play data, player participation data, and analysisdata. The analysis data file includes data file partitions ofuser-editable data fields for each play of the game. In the exampleembodiment, AE computing device 110 determines a type of play (e.g.,passing, running, blown play, special teams, etc.) from the base playdata for at least some of the plays and updates the analysis data fileto include contextual data fields specific to the determined type ofplay. In one example, AE computing device 110 analyzes the base playdata for play type identifiers to determine the play type. In anotherexample, AE computing device 110 automatically determines a play typebased on indirect information, such as other plays or indicators. In oneexample, AE computing device 110 automatically determines a kick-offfollows after a point-after-attempt (PAT) or halftime.

Once task scores have been assigned for a play for each participatingplayer, the task scores are normalized by determining one or morenormalization factors representative of game conditions for the taskfrom the analysis data file for each task. The normalization factors maybe determined, for example, from the stored base play data, playerparticipation data, category identifiers, and analysis data of theanalysis data file, such as down and distance, quarterback drop depth(e.g., three steps or seven steps), and pressure applied to thequarterback. AE computing device 110 is configured to use thenormalization factors to locate historical task scores that match thenormalization factors. The historical task scores are averaged togetherto generate a normalization model data. The normalization model data isapplied to the task score during normalization to generate a normalizedtask score. In some embodiments, a replacement factor is also determinedfor a replacement level player and applied to the task score to generatethe normalized task score.

For example on a three-step drop for a quarterback, it may be difficultfor a pass rusher to get a positive task score. On a seven-step drop, itis comparatively easier to get a positive pass rush task score, so thesegame conditions are normalized differently. On a play with a seven-stepdrop, a pass rusher on average gets a +0.10 task score, and on a playwith a three-step drop, a pass rusher on average gets a +0.04 taskscore. If pass rushers were normalized just based on the drop depth ofthe quarterback, 0.10 from their pass rush task scores would besubtracted for every seven-step drop that the pass rushers participatedin, and 0.04 would be subtracted for every three-step drop that the passrushers participated in.

Similarly, it may be relatively easier to get defensive pressure at somepositions compared to others. In one example, a nose tackle averages apass rush task score of +0.049 per pass rush while a middle linebackeraverages +0.078 per pass rush. If normalized just by position, 0.049 issubtracted from a pass rush task score at the nose tackle position and0.078 is subtracted from a pass rush task score at the middle linebackerposition. By combining multiple normalization factors, such as playerposition and quarterback drop depth, normalization of task scores usinga normalization model data facilitates normalizing the player'sperformance relative to historical player performance in similar playconditions, thereby enhancing performance evaluation.

In the example embodiment, the normalized task scores are averaged overa corresponding number of plays a player participated in to generate aperformance grade for the player. In some embodiments, the performancegrade is converted into a different scale or format for display. Forexample, the converted performance score may rate players on a 0-100scale. AE computing device 110 stores the normalized task scores and theperformance grades with the analysis data file. In at least someembodiments, the performance grades may be used to calculate a season orcareer performance grade for each player. That is, performance gradesfor a player from a plurality of games are used to calculate a season orcareer performance grade for the player.

In the example embodiment, AE computing device 110 is configured totransmit the performance grades to one or more client computing devices(e.g., 132) associated with end users to cause the performance grades tobe displayed to the end users. The end users may use the performancegrades to influence decisions, such as fantasy football transactions,contract negotiation, and player evaluation. In at least someembodiments, AE computing device 110 transmits the analysis data file toan end user. The analysis data file may be added to the metadata of thevideo data file to facilitate navigation within the video data file.

In some embodiments, the analysis data file is stored in one or moreanalysis data packages (not shown) by AE computing device 110. Theanalysis data packages represent a plurality of games and include aplurality of analysis data files. In some embodiments, the analysis datapackages include portions of analysis data files (e.g., an analysis datapackage associated with a particular player). In one example, theanalysis data package may represent a season or a player's career. Theanalysis data package is used by a client computing device to enablesearching for and filtering data within the stored analysis data files.In at least some embodiments, AE computing device 110 is configured toidentify trends or other analytics for the analysis data package. Theidentified trends may include trends in performance grades, task scores,game data, and any other data stored in the analysis data files. Theseidentified trends may be provided to an end user for review. In certainembodiments, the analysis data package is linked to one or more eventdata files to enable an end user to navigate the event data files usingthe analysis data package.

FIG. 9 illustrates an example user interface 900 used by AE system 100(shown in FIG. 1 ) to input base data associated with a football game.AE system 100 is configured to display user interface 900 on computingdevices associated with real-time data sources 142, 144, and 146 (allshown in FIG. 1 ) and collect from user interface 900 the base data.User interface 900 receives the base data and displays the base data.User interface 900 includes a pre-snap entry section 902 and a map entrysection 904. Pre-snap entry section 902 and map entry section 904 may beused to input data into user interface 900. Pre-snap entry section 902includes a sequence of a task 906, a game clock 908, a team possession910, a drive 912, a play 914, a field position 916, a quarter 918, apossession down 920, a distance from the line to gain 922. Map entrysection 904 includes a map of a football field 924 and statisticssection 926. User interface 900 also includes a task tracker 928 and apost-snap entry section 930. Map of the football field 924 may be aninteractive graphical user interface (GUI) that includes a graphics in aplurality of colors (e.g., color-coded graphics). Each of the graphicsin the plurality of colors may represent players, yards on the footballfield, a line of scrimmage, the line to gain, a distance of throw, adistance of a run, and/or other data that may input and/or output fromuser interface 900. AE system may update map of the football field 924by using data input from pre-snap entry section 902, map entry section904, task tracker 928, and a post-snap entry section 930.

In an example, user interface 900 receives and displays base dataassociated with a football game between Chicago (CHI) and Houston (HST).Task tracker 928 displays that task 2 of the football game was a firstdown possession at 10 yards from the line to gain, and morespecifically, at the 27-yard of HST's side of the football field. Tasktracker 928 also displays that task 2 was performed at 14 minutes and 54seconds remaining in a first quarter of the football game. Post-snapentry section 930 includes one or more drop downs 932, one or more checkboxes 934, and one or more entry fields 936. Task tracker 928 is incommunication with pre-snap entry section 902, map entry section 904,and post-snap entry section 930. Continuing with the above example, mapentry section 904 displays, in football field 924, at least one player938 at the 27-yard of HST's side of the football field. Map entrysection 904 also displays, in statistics section 926, a number 940associated with the at least one player 938 performing task 2 and atleast one position 942 of the at least one player 938.

FIG. 10 is an example user interface 1000 used by AE system 100 (shownin FIG. 1 ) to input base data associated with a football game. Userinterface 1000 is similar to user interface 900 (shown in FIG. 9 ) andincludes the data fields of user interface 900. User interface 1000 alsoincludes a task measurement section 1002. Task measurement section 1002includes the following data fields: away team player number 1004, awayteam player rating 1006, away player play 1008, home team player number1010, home team player rating 1012, home team player play 1014, andcomment field 1016. A real-time source, such as real-time data sources142, 144, and 146 (all shown in FIG. 1 ) inputs data (i.e., taskmeasurement data and base data) into the data fields of task measurementsection 1002. AE system 100, and more specifically, AE computing device110 (shown in FIG. 1 ) is configured to receive the task measurementdata and the base data from user interface 1000 and store the taskmeasurement data and base data within a database (e.g., MySQL database).AE computing device 110 parses the task measurement data and base data,and identifies and assigns at least one category identifier to the taskmeasurement data based on the parsed task measurement and base data.Category identifiers may be associated with the following categories:Passing, Rushing, Pass Blocking, Run Blocking, Receiving, Screen Block,Offensive Penalty, Pass Rushing, Run Defense, Coverage, DefensivePenalty, Kickoff Penalties, Kicking, Kick Return, Kickoffs, PuntPenalties, Punting, Punt Returns, Punts, Field Goal Penalties, FieldGoal, Field Goals, Pass Defense, and Scramble.

FIGS. 11A and 11B are examples of user interfaces 1100 and 1102,respectively, used by AE system 100 (shown in FIG. 1 ) to displayparties (e.g., players) participation in a multi-interaction (e.g.,event or game). With reference to FIG. 11A, user interface 1100 displaysa number of plays (e.g., events) for each player 1104 (e.g., party), aplayer identifier 1106 which includes a player name and jersey number.In some embodiments, player identifier 1106 further includes partynotes, such as information to distinguish the player from other playerson the field. In addition, user interface 1100 displays a playerposition 1108, a stance indicator 1110, a player's primary role on thegiven task 1112 (e.g., category identifier), line of scrimmage (LOS)check box 1114, injury check box 1116, a team identifier 1118, and anotes field 1119. Notes field 1119 includes additional interaction dataassociated with the player and/or play (e.g., interaction addendumdata).

With reference to FIG. 11B, user interface 1102 displays a playeridentifier 1120, role on the play 1142, a stance indicator 1122, an LOSindicator 1124, and a player injury indicator 1126. In addition, userinterface 1102 includes data fields associated with offensive players,such as a starting and/or ending pre-snap position 1128, width 1130,depth 1132, and motion 1140. In certain embodiments, AE computing device110 is configured to identify pre-snap shifts in player formations anddetermine if a player is in motion as the ball is snapped. In oneembodiment, AE computing device 110 identifies pre-snap shifts (e.g.,interactions) based at least in part on pre-snap position 1128, width1130, depth 1132, and motion 1140. For example, AE computing device 110may determine that a player moved from the edge of the field towards thecenter of the field in a pre-snap shift. User interface 1102 includesdata fields associated with defensive players, such as position 1128,depth 1132, technique 1134, width 1130, and press coverage indicator1141. Technique 1134 refers to the alignment of a player on thedefensive line against their offensive line counterpart and includes apress coverage indicator 1141. User interface 1102 also includes a teamidentifier 1136, and a notes field 1138. User interfaces 1100 and 1102enable real-time data sources to input player participation (PP) datainto AE system 100. More specifically, some of the data displayed inuser interface 1102 is input from user interface 1100 and may enable areal-time data source to verify the input from user interface 1100.

AE system 100, and more specifically, AE computing device 110 (shown inFIG. 1 ) is configured to build a matrix of players by categoryidentifier, such as the category identifiers associated with thecategories listed in FIG. 10 . For example, AE computing device 110 mayfill the matrix with 0's such that if a player was in coverage on agiven play that all other tasks are null but coverage is 0. 0 is thebase task measurement for a given player. All players start at 0 on agiven play.

Subsequently, AE computing device 110 is configured to overlay the taskthat has assigned at least one category identifier (described in FIG. 10) on the data in the matrix. For example, if the PP data indicates thata player is in coverage, but the player actually also got a pass rushtask measurement, AE computing device 110 is configured to assign toboth the coverage and pass rush measurements their respectivecategories. The AE computing device is also configured to add orsubtract to the task measurements with assigned categories anormalization factor based on normalization model data.

AE computing device 110 is further configured to store for each player araw task measurement, a factor number associated with the normalizationfactor, and a computation of a normalized task score, which is anaddition of the raw task measurement and the factor number. For eachplayer in each game, AE computing device 110 adds the normalized taskscore per task and stores the normalized task scores within a table(e.g., a normalized table within normalization database 121 asillustrated in FIG. 2 ) such that a player may have a normalized taskscore across any number of tasks during a multi-interaction (e.g., agame) and/or numerous multi-interaction (e.g., a partial or fullseason). AE computing device 110 is also configured to retrieve, perplayer per task, the normalized task scores from the normalized tableand compare the normalized task score to an average of normalized taskscore for the same task in order to generate a ranking of 0-100 of theplayer.

In some embodiments, AE computing device 110 is configured to collectbase data and task measurement data once every 10th of a second for allplayers whether the player are in the field or in the sideline. AEcomputing device 110 is also configured to filter the base data and thetask measurement data to more efficiently compute the normalized scores.AE computing device 110 is further configured to receive height data(distance from the ground) to determine a players' stances ortechniques. AE computing device 110 is also configured to use base datafrom pre-snap entry section 902 (shown in FIG. 9 ) to determine a shift(e.g., movement when a ball is stationary as the ball is snapped)through to motion (e.g., receiver motion as the ball is snapped). AEcomputing device 110 may use the base data from pre-snap entry section902 to identify shifting in, for example, defensive patterns (e.g.,reaction to the movements in the offensive line).

AE computing device 110 is also configured to determine a plurality ofroutes, for example, the routes a receiver ran and to gauge aspects ofthe run, such as speed of the receiver and receiver's running strategiesand techniques. For example, AE computing device 110 may generate aresponse to questions, such as “Does the receiver break to the inside oroutside? Does the movement of the receiver correlate with the receiver'spre-snap position? If receiver A does X, does another receiver Y do B?Does receiver X have more success against defender J if he breaks in acertain direction?” AE computing device is further configured toidentify the separation between a receiver and defenders in coverage ofthe receiver, a defensive scheme (e.g., man, zone—in other words, did adefender track a receiver as he moved pre-snap or did?), how well anoffensive line did in pass protection, how well was the pocket protected(e.g., did the pocket collapse in a certain way (left side, right side),how long did they hold that protection for?), the correlation of theprotected pocket and the task measurements that the offensive line getin pass blocking and the defensive line get in pass rushing.

The user interfaces described herein include some examples of how theuser interfaces may be displayed and input data. These examples are notintended to limit the data display and input of the user interfaces inany way. Rather, these user interface are used to display and inputinteractive data for numerous multi-party interactions. For example, arun play may be displayed differently in the map entry section than apass play in the same section.

In the example embodiment, real-time data source refers to a computingdevice associated with a collector, and a client computing device refersto a computing device associated with an end-user. For example, acollector may observe a football game and record task measurements usinga computing device, and an end-user may retrieve analysis data using theclient computing device.

In the example embodiment, where the AE computing device is configuredto analyze a football game, event refers to a football game, interactionrefers to a football play, party refers to a football player, and taskrefers to a play activity of a player (i.e., a player's actions) withina specific football play. Additionally, a category may be associatedwith a football play or a football player indicating a player's positionor play type. In certain embodiments, addendum data (e.g., task addendumdata, interaction addendum data) may be stored based on categoriesincluding player positions and play types.

In the example embodiment, task measurement data refers to a specificdata point input by a collector regarding a football player, such as ayard measurement or player location, and contextual data refers toinformation including a player roster. Task score refers to a scoreassociated with a play activity (e.g., task), such as blocking orpassing. Interaction data may include any combination of taskmeasurement data and task score data, such as a set of measurementsrepresenting all player activity within a football play.

In the example embodiment, normalization rules include determining anormalization factor associated with a task score, such as determining anormalization factor based on the type of play and the position of theplayer, and normalization factors include adjusting task scores based onaggregates of comparable task scores. For example, comparable taskscores may include task scores having a similar player position and/ortype of play.

Output analysis data includes, in some embodiments, score data such asnormalized task scores and aggregate scores. In the example embodiment,output analysis includes normalized task scores representing a score ofa player's action, such as a blocking or passing tasks, where the scoreis normalized relative to comparable task scores, and aggregate scoresinclude an overall player score, such as a player ranking for a game, aseries of games, and/or an entire season (e.g., a football season).

As will be appreciated based on the foregoing specification, theabove-discussed embodiments of the disclosure may be implemented usingcomputer programming or engineering techniques including computersoftware, firmware, hardware or any combination or subset thereof. Anysuch resulting computer program, having computer-readable and/orcomputer-executable instructions, may be embodied or provided within oneor more computer-readable media, thereby making a computer programproduct, i.e., an article of manufacture, according to the discussedembodiments of the disclosure. These computer programs (also known asprograms, software, software applications, or code) include machineinstructions for a programmable processor, and can be implemented in ahigh-level procedural and/or object-oriented programming language,and/or in assembly/machine language. As used herein, the terms“machine-readable medium,” “computer-readable medium,” and“computer-readable media” refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The “machine-readable medium,” “computer-readable medium,” and“computer-readable media,” however, do not include transitory signals(i.e., they are “non-transitory”). The term “machine-readable signal”refers to any signal used to provide machine instructions and/or data toa programmable processor.

This written description uses examples to disclose the invention,including the best mode, and also to enable any person skilled in theart to practice the invention, including making and using any devices orsystems and performing any incorporated methods. The patentable scope ofthe invention is defined by the claims, and may include other examplesthat occur to those skilled in the art. Such other examples are intendedto be within the scope of the claims if they have structural elementsthat do not differ from the literal language of the claims, or if theyinclude equivalent structural elements with insubstantial differencesfrom the literal languages of the claims.

What is claimed is:
 1. An analytics engine (AE) computing system foranalyzing and evaluating data in real-time associated with a performanceof parties interacting within a multi-party interaction, the AEcomputing system comprising at least one analytics engine (AE) computingdevice comprising a processor and a memory communicatively coupled tothe processor, the processor programmed to: generate a normalizationmodel based on a plurality of normalization rules; store thenormalization model in a normalization database; update thenormalization model with one or more normalization factors;electronically receive, from a data validation (DV) computing device,validated interaction data of the multi-party interaction including atleast a real-time data source identifier, a party identifier, taskmeasurement data, and at least one category identifier, wherein themulti-party interaction includes a plurality of interactions, whereinthe validated interaction data is generated from real-time video data ofthe multi-party interaction; identify a first party identifierassociated with a first interaction of the plurality of interactions,wherein the first party identifier includes a position associated with afirst party during the first interaction; retrieve contextual data froma contextual data source based on the first party identifier in thevalidated interaction data, wherein the contextual data includes atleast an interaction identifier associated with the first interaction;execute the normalization model with the validated interaction data andthe contextual data to determine at least one normalization factor toapply to a task score of the first party, wherein the at least onenormalization factor is based on the position associated with the firstparty and the at least one category identifier for the firstinteraction; calculate an aggregate score based on the at least onenormalization factor; and generate a display of one or more aggregatescores associated with one or more parties in the multi-partyinteraction.
 2. The AE computing system of claim 1, wherein theprocessor further programmed to: normalize the task score based on theat least one normalization factor; and calculate an aggregate scoreusing the normalized task score.
 3. The AE computing system of claim 2,wherein the processor further programmed to store the validatedinteraction data, the normalized task score, and the aggregate score inan analysis database based on a task identifier, wherein the analysisdatabase is partitioned based at least in part on a party identifier anda task identifier.
 4. The AE computing system of claim 3, wherein theanalysis database includes a task table indexed using task identifiers,an interaction table indexed using interaction identifiers, and a partytable indexed using party identifiers.
 5. The AE computing system ofclaim 4, wherein the processor further programmed to map, using thenormalization rules, the category identifiers to one or morenormalization factors based on the party identifiers.
 6. The AEcomputing system of claim 5, wherein storing the validated interactiondata further includes generating database instructions configured tocreate a task record in the task table based on the determined taskidentifier, and the database instructions further include the validatedinteraction data, the normalized task score, and the aggregate score. 7.The AE computing system of claim 1, wherein the task identifier includesa combination of the interaction identifier and the party identifier. 8.The AE computing system of claim 1, wherein the processor furtherprogrammed to: analyze the validated interaction data; determine amissing task measurement not included in the validated interaction data;and transmit a notification message to at least one of a real-time datasource and the DV computing device, wherein the notification messageincludes a task measurement identifier associated with the missing taskmeasurement.
 9. The AE computing system of claim 1, wherein theprocessor further programmed to: receive a first task score for a firsttask identifier from a first data source; receive a second task scorefor the first task identifier from a second data source; compare thefirst task score and the second task score; if the first task score andthe second task score match, verify and store the first task score; andif the first task score and the second task score do not match, identifyand store one or more differences based on the comparison.
 10. Acomputer-implemented method for analyzing and evaluating data inreal-time associated with a performance of parties interacting within amulti-party interaction, the method implemented using analytics engine(AE) computing device in communication with a memory, the methodcomprising: generating a normalization model based on a plurality ofnormalization rules; storing the normalization model in a normalizationdatabase; updating the normalization model with one or morenormalization factors; electronically receiving, from a data validation(DV) computing device, validated interaction data of the multi-partyinteraction including at least a real-time data source identifier, aparty identifier, task measurement data, and at least one categoryidentifier, wherein the multi-party interaction includes a plurality ofinteractions, wherein the validated interaction data is generated fromreal-time video data of the multi-party interaction; identifying a firstparty identifier associated with a first interaction of the plurality ofinteractions, wherein the first party identifier includes a positionassociated with a first party during the first interaction; retrievingcontextual data from a contextual data source based on the first partyidentifier in the validated interaction data, wherein the contextualdata includes at least an interaction identifier associated with thefirst interaction; execute the normalization model with-the validatedinteraction data and the contextual data to determine at least onenormalization factor to apply to a task score of the first party,wherein the at least one normalization factor is based on the positionassociated with the first party and the at least one category identifierfor the first interaction; calculating an aggregate score based on theat least one normalization factor; and generating a display of one ormore aggregate scores associated with one or more parties in themulti-party interaction.
 11. The method of claim 10 further comprising:normalizing the task score based on the at least one normalizationfactor; and calculating an aggregate score using the normalized taskscore.
 12. The method of claim 11 further comprising storing thevalidated interaction data, the normalized task score, and the aggregatescore in an analysis database based on a task identifier, wherein theanalysis database is partitioned based at least in part on a partyidentifier and a task identifier.
 13. The method of claim 12, whereinthe analysis database includes a task table indexed using taskidentifiers, an interaction table indexed using interaction identifiers,and a party table indexed using party identifiers.
 14. The method ofclaim 13 further comprising mapping, using the normalization rules, thecategory identifiers to one or more normalization factors based on theparty identifiers.
 15. The method of claim 13, wherein storing thevalidated interaction data further comprises generating databaseinstructions configured to create a task record in the task table basedon the determined task identifier, and the database instructions furtherinclude the validated interaction data, the normalized task score, andthe aggregate score.
 16. The method of claim 10, wherein the taskidentifier includes a combination of the interaction identifier and theparty identifier.
 17. The method of claim 10 further comprising:analyzing the validated interaction data; determining a missing taskmeasurement not included in the validated interaction data; andtransmitting a notification message to at least one of a real-time datasource and the DV computing device, wherein the notification messageincludes a task measurement identifier associated with the missing taskmeasurement.
 18. The method of claim 10 further comprising: receiving afirst task score for a first task identifier from a first data source;receiving a second task score for the first task identifier from asecond data source; comparing the first task score and the second taskscore; if the first task score and the second task score match,verifying and storing the first task score; and if the first task scoreand the second task score do not match, identifying and storing one ormore differences based on the comparison.
 19. A non-transitorycomputer-readable storage media having computer-executable instructionsembodied thereon, wherein when executed by an analytics engine (AE)computing device having at least one processor coupled to at least onememory device, the computer-executable instructions cause the processorto: generate a normalization model based on a plurality of normalizationrules; store the normalization model in a normalization database; updatethe normalization model with one or more normalization factors;electronically receive, from a data validation (DV) computing device,validated interaction data of a multi-party interaction including atleast a real-time data source identifier, a party identifier, taskmeasurement data, and at least one category identifier, wherein themulti-party interaction includes a plurality of interactions, whereinthe validated interaction data is generated from real-time video data ofthe multi-party interaction; identify a first party identifierassociated with a first interaction of the plurality of interactions,wherein the first party identifier includes a position associated with afirst party during the first interaction; retrieve contextual data froma contextual data source based on the first party identifier in thevalidated interaction data, wherein the contextual data includes atleast an interaction identifier associated with the first interaction;execute the normalization model with-the validated interaction data andthe contextual data to determine at least one normalization factor toapply to a task score of the first party, wherein the at least onenormalization factor is based on the position associated with the firstparty and the at least one category identifier for the firstinteraction; calculate an aggregate score based on the at least onenormalization factor; and generate a display of one or more aggregatescores associated with one or more parties in the multi-partyinteraction.
 20. The computer-executable instructions of claim 19, thecomputer-executable instructions further cause the processor to: analyzethe validated interaction data; determine a missing task measurement notincluded in the validated interaction data; and transmit a notificationmessage to at least one of a real-time data source and the DV computingdevice, wherein the notification message includes a task measurementidentifier associated with the missing task measurement.